Metrically Scaled Monocular Depth Estimation through Sparse Priors for Underwater Robots

In this work, we address the problem of real-time dense depth estimation from monocular images for mobile underwater vehicles. We formulate a deep learning model that fuses sparse depth measurements from triangulated features to improve the depth predictions and solve the problem of scale ambiguity. To allow prior inputs of arbitrary sparsity, we apply a dense parameterization method. Our model extends recent state-of-the-art approaches to monocular image based depth estimation, using an efficient encoder-decoder backbone and modern lightweight transformer optimization stage to encode global context. The network is trained in a supervised fashion on the forward-looking underwater dataset, FLSea. Evaluation results on this dataset demonstrate significant improvement in depth prediction accuracy by the fusion of the sparse feature priors. In addition, without any retraining, our method achieves similar depth prediction accuracy on a downward looking dataset we collected with a diver operated camera rig, conducting a survey of a coral reef. The method achieves real-time performance, running at 160 FPS on a laptop GPU and 7 FPS on a single CPU core and is suitable for direct deployment on embedded systems. The implementation of this work is made publicly available at https://github.com/ebnerluca/uw_depth.Comment: Submitted to ICRA 202

Hybrid Visual SLAM for Underwater Vehicle Manipulator Systems

This paper presents a novel visual scene mapping method for underwater vehicle manipulator systems (UVMSs), with specific emphasis on robust mapping in natural seafloor environments. Prior methods for underwater scene mapping typically process the data offline, while existing underwater SLAM methods that run in real-time are generally focused on localization and not mapping. Our method uses GPU accelerated SIFT features in a graph optimization framework to build a feature map. The map scale is constrained by features from a vehicle mounted stereo camera, and we exploit the dynamic positioning capability of the manipulator system by fusing features from a wrist mounted fisheye camera into the map to extend it beyond the limited viewpoint of the vehicle mounted cameras. Our hybrid SLAM method is evaluated on challenging image sequences collected with a UVMS in natural deep seafloor environments of the Costa Rican continental shelf margin, and we also evaluate the stereo only mode on a shallow reef survey dataset. Results on these datasets demonstrate the high accuracy of our system and suitability for operating in diverse and natural seafloor environments.Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessibl

Towards automated sample collection and return in extreme underwater environments

© The Author(s), 2022. This article is distributed under the terms of the Creative Commons Attribution License. The definitive version was published in Billings, G., Walter, M., Pizarro, O., Johnson-Roberson, M., & Camilli, R. Towards automated sample collection and return in extreme underwater environments. Journal of Field Robotics, 2(1), (2022): 1351–1385, https://doi.org/10.55417/fr.2022045.In this report, we present the system design, operational strategy, and results of coordinated multivehicle field demonstrations of autonomous marine robotic technologies in search-for-life missions within the Pacific shelf margin of Costa Rica and the Santorini-Kolumbo caldera complex, which serve as analogs to environments that may exist in oceans beyond Earth. This report focuses on the automation of remotely operated vehicle (ROV) manipulator operations for targeted biological sample-collection-and-return from the seafloor. In the context of future extraterrestrial exploration missions to ocean worlds, an ROV is an analog to a planetary lander, which must be capable of high-level autonomy. Our field trials involve two underwater vehicles, the SuBastian ROV and the Nereid Under Ice (NUI) hybrid ROV for mixed initiative (i.e., teleoperated or autonomous) missions, both equipped seven-degrees-of-freedom hydraulic manipulators. We describe an adaptable, hardware-independent computer vision architecture that enables high-level automated manipulation. The vision system provides a three-dimensional understanding of the workspace to inform manipulator motion planning in complex unstructured environments. We demonstrate the effectiveness of the vision system and control framework through field trials in increasingly challenging environments, including the automated collection and return of biological samples from within the active undersea volcano Kolumbo. Based on our experiences in the field, we discuss the performance of our system and identify promising directions for future research.This work was funded under a NASA PSTAR grant, number NNX16AL08G, and by the National Science Foundation under grants IIS-1830660 and IIS-1830500. The authors would like to thank the Costa Rican Ministry of Environment and Energy and National System of Conservation Areas for permitting research operations at the Costa Rican shelf margin, and the Schmidt Ocean Institute (including the captain and crew of the R/V Falkor and ROV SuBastian) for their generous support and making the FK181210 expedition safe and highly successful. Additionally, the authors would like to thank the Greek Ministry of Foreign Affairs for permitting the 2019 Kolumbo Expedition to the Kolumbo and Santorini calderas, as well as Prof. Evi Nomikou and Dr. Aggelos Mallios for their expert guidance and tireless contributions to the expedition

Visual Methods Towards Autonomous Underwater Manipulation

Extra-terrestrial ocean worlds like Europa offer tantalizing targets in the search for extant life beyond the confines of Earth's atmosphere. However, reaching and exploring the underwater environments of these alien worlds is a task with immense challenges. Unlike terrestrial based missions, the exploration of ocean worlds necessitates robots which are capable of fully automated operation. These robots must rely on local sensors to interpret the scene, plan their motions, and complete their mission tasks. Manipulation tasks, such as sample collection, are particularly challenging in underwater environments, where the manipulation platform is mobile, and the environment is unstructured. This dissertation addresses some of the challenges in visual scene understanding to support autonomous manipulation with underwater vehicle manipulator systems (UVMSs). Specifically, this work addresses the problems of tool detection and pose estimation, 3D scene reconstruction, underwater camera system design, underwater dataset collection, and UVMS manipulator automation. The developed visual methods are demonstrated with a lightweight vision system, composed of a vehicle mounted stereo pair and a manipulator wrist mounted fisheye camera, that can be easily integrated on existing UVMSs. While the stereo camera primarily supports 3D reconstruction of the manipulator working area, the wrist mounted camera enables dynamic viewpoint acquisition for detecting objects, such as tools, and extending the scene reconstruction beyond the fixed stereo view. A further objective of this dissertation was to apply deep learning with the developed visual methods. While deep learning has greatly advanced the state-of-the-art in terrestrial based visual methods across diverse applications, the challenges of accessing the underwater environment and collecting underwater datasets for training these methods has hindered progress in advancing visual methods for underwater applications. Following is an overview of the contributions made by this dissertation. The first contribution is a novel deep learning method for object detection and pose estimation from monocular images. The second contribution is a general framework for adapting monocular image-based pose estimation networks to work on full fisheye or omni-directional images with minimal modification to the network architecture. The third contribution is a visual SLAM method designed for UVMSs that fuses features from both the wrist mounted fisheye camera and the vehicle mounted stereo pair into the same map, where the map scale is constrained by the stereo features, and the wrist camera can actively extend the map beyond the limited stereo view. The fourth contribution is an open-source tool to aid the design of underwater camera and lighting systems. The fifth contribution is an autonomy framework for UVMS manipulator control and the vision system that was used throughout this dissertation work, along with experimental results from field trials in natural deep ocean environments, including an active submarine volcano in the Mediterranean basin. The sixth contribution is a large scale annotated underwater visual dataset for object pose estimation and 3D scene reconstruction. The dataset was collected with our vision system in natural deep ocean environments and supported the development of the visual methods contributed by this dissertation.PHDRoboticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/172545/1/gidobot_1.pd